Skip to content

fix(writer): bit-packing and ALP write their patches as Rust does - #619

Merged
dfa1 merged 1 commit into
mainfrom
fix/bitpacked-patch-chunk-offsets
Oct 31, 2026
Merged

dfa1 merged 1 commit into
mainfrom
fix/bitpacked-patch-chunk-offsets

Conversation

@dfa1

@dfa1 dfa1 commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Follow-up to #609, which left patched columns out of its vortex-jni cross-checks: Rust stores bit-packing patches with chunk offsets and Java did not.

Two commits, one per bug:

  1. Bit-packing patches. Rust's gather_patches records the number of patches before each 1024-value chunk, and the bit-packing scheme runs compress_patches (vortex-btrblocks schemes/patches.rs). That narrows the indices and chunk offsets to the smallest unsigned type that holds them, and stores equal values as a vortex.constant. Java wrote the indices (typed by array length) and primitive values only. The patch layout now goes through PatchBuffers, which ALP already used.
  2. ALP patch values. Rust's ALP scheme runs the same compress_patches, so equal exceptions become a vortex.constant. Java always wrote a primitive. ALP's four patch sites now use the same PatchBuffers.append.

Tests

  • NullableIntegerInteropIntegrationTest: wide outliers and equal outliers shapes (patched bit-packing). Each asserts vortex-jni's encoding tree, then that vortex-jni reads Java's file back (default and compact; full, filtered, row-range). Both tree checks fail without commit 1.
  • NullableFloatInteropIntegrationTest: f64 equal exceptions and f32 equal exceptions. Both fail without commit 2.
  • Unit tests: narrowed indices and chunk offsets, constant patch values, and round trips for bit-packing and ALP.
  • ./mvnw verify is green locally.

🤖 Generated with Claude Code

Rust's gather_patches (fastlanes bitpack_compress/global.rs) records, for each 1024-value
chunk, how many patches come before it, and the bit-packing and ALP schemes then run
compress_patches (vortex-btrblocks schemes/patches.rs): indices and chunk offsets narrowed
to the smallest unsigned type that holds their largest value, and values that are all
equal (one value included) stored as a vortex.constant. Java wrote bit-packing's indices
typed by the array length, no chunk offsets, and both encodings' values as a primitive,
so a patched column held a different encoding tree from vortex-jni's.

The patch layout of bit-packing, ALP and ALP-RD now goes through PatchBuffers;
PatchBuffers.valuesNode is the constant-or-primitive rule, checking and encoding in one
scan (ConstantEncodingEncoder.encodeIfConstant). Bit-packing hands its widened patch
values straight through, written at their width in one copy, and ALP collects its
exceptions into primitive arrays instead of boxed lists.

Tests: NullableIntegerInteropIntegrationTest covers patched columns (wide and equal
outliers), which #609 had to leave out, and NullableFloatInteropIntegrationTest F64 and
F32 shapes with equal exceptions, with and without nulls; all fail without the change.
Bit-packing patches round-trip for every integer ptype (U8 200 and U64 above 2^63 catch a
sign-extension); a wide value at a null row is never patched, directly or through FoR
over negatives; PatchBuffersTest covers chunk-boundary offsets and narrowing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dfa1
dfa1 force-pushed the fix/bitpacked-patch-chunk-offsets branch from 9360965 to d15d023 Compare October 31, 2026 14:48
@dfa1
dfa1 merged commit a6e2ded into main Oct 31, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/bitpacked-patch-chunk-offsets branch November 1, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant